Thanks! I've addressed these comments and found another place the lowercase mapfs was used instead of MAPFS.

When a file is mapped (e.g. /path/whatever -> /fetchfs/server.com/url/path/file) what is the relationship between the source the destination file? Is it like a hard link? Or a sym link? Will opening both the source and the destination at the same time result in a file with same inode number?

When a file is mapped (e.g. /path/whatever -> /fetchfs/server.com/url/path/file) what is the relationship between the source the destination file? Is it like a hard link? Or a sym link? Will opening both the source and the destination at the same time result in a file with same inode number?

A "virtual data file" is created, as in the icase backend, which refers to the original file. It is different from icase in that the original file can also be accessed by some path.

What should happen/what bad things could happen/what test cases should be added to explore the possibility of the file being opened under two different names?

When a file is mapped (e.g. /path/whatever -> /fetchfs/server.com/url/path/file) what is the relationship between the source the destination file? Is it like a hard link? Or a sym link? Will opening both the source and the destination at the same time result in a file with same inode number?

A "virtual data file" is created, as in the icase backend, which refers to the original file. It is different from icase in that the original file can also be accessed by some path.

What should happen/what bad things could happen/what test cases should be added to explore the possibility of the file being opened under two different names?

Would symlinks be a simpler/quicker solution?

Should we add hard link support maybe?

We thought about symlinks before, but this will work even if the C code doesn't do the right thing wrt symlinks. One could argue that such C code should be fixed instead of having a new mapping backend, but it becomes complicated with e.g. whether writes through remapping should modify the location of the symlink or the underlying target location (perhaps not relevant for the fetch backend but maybe relevant in other settings).

We thought about symlinks before, but this will work even if the C code doesn't do the right thing wrt symlinks. One could argue that such C code should be fixed instead of having a new mapping backend, but it becomes complicated with e.g. whether writes through remapping should modify the location of the symlink or the underlying target location (perhaps not relevant for the fetch backend but maybe relevant in other settings).

Fair enough, symlink trees are not the same as normal files.

How about hard links? @tlively @kripken did you even think of adding support for the same file existing in multiple places? (or more than one file having the same inode number?)

No, I don't think we ever seriously considered supporting hard links. Backends like OPFS would not be able to support them because they could not be reflected in the underlying storage. I don't have enough of the system paged into memory to say what other specific problems might arise from allowing files to be reachable from multiple paths, but I am very wary of the idea.

Does this backend make the same file visible from multiple paths, or does it virtualize and provide a mapped view into some underlying backend that is not directly mounted into the file system? I couldn't tell from a quick look at the code. I would strongly prefer the latter design to avoid making a file directly accessible from multiple paths.

This does the former, but I think it’s a small change to make it do the latter:

1. Take a backend parameter
2. Use createFile / createDirectory on the inner backend at the same time as they are created on the mapfs backend.

This would make it more like the icase backend, so I’ll give that a shot over the next day or two.

So I think it's not actually going to be possible with the current API to define mapfs as a front to another, non-mounted filesystem, especially if the wasmfs_create_file API is going away, because intermediate directories &c may need to be created in the non-mounted filesystem. The backend would need specialized API that takes both a real path on the wasmfs filesystem and a virtual path on the mapped-to filesystem backend. Regular open(O_CREAT)/mkdir type calls won't work to create files in mapfs without a much more sophisticated mapping than a simple manifest.

Not every filesystem backend allows writes, and already "creating" a file in the fetch backend (for example) is an awkward idea (since files are readonly)---moreover, doMkdir and doOpen also "mount" the created files on the filesystem, so the syscalls both create and mount. So I believe the best solution is to withdraw mapfs and require that C programs handle symlinks properly, and also make fetchfs automatically create files on open (even without O_CREAT)---but even that might not actually be possible, since getChild() can't know if it's getting a data file, directory, or something else.

So as I see it, the manifest makes sense as part of fetchfs (mapping filesystem paths and URLs, designating actually accessible files) but has big question marks when implemented as a standalone filesystem. So I think a standalone mapfs has too many rough edges and shouldn't be implemented; either fetchfs should use a manifest (and avoid the need for "creating"/opening fetched files) or we should just require that C programs do the right thing with symlinks.

Thanks for that analysis, @JoeOsborn. It sounds right to me. I would be fine with adding manifests to fetchfs if there is demand for it, but I would also hope that most programs handle symlinks correctly.

I’ll close this as a successful experiment.